Project 2 Board game object tracking

• Krzysztof Weber 151934
• Adam Mielniczuk 151928

The game chosen by us is chess. The aim of the first milestone was to detect two game peaces. We were able to detect a chessboard and all the peaces. There are still some difficulties with the object detection but we are still working on it.

All functions used in latter examples are defined in utils.py file. It is to make the report more readable.

from utils import *

Data preparation.

We have prepared 9 recordings of 9 chess games. The are in three categories:

1. Easy - Static camera, good lighting, hand covering parts of the board
2. Medium - Static camera, shady lighting, hand covering parts of the board
3. Hard - Shaky camera movements, shady lighting, hand covering parts of the board

Board detection

chess = video_from_path('recordings\Easy1.mp4')

Video loaded
912 720
30.00030000300003

frame = get_first_frame(chess)

Finding lines is made using HoughLinesP function along with canny edge detection. The lines are then represented with slopes and shifts defined in the form of y = ax + b and x = ay + b due to the fact that many lines are perfectly or almost perfectly vertical or horizontal.

lines = find_lines(frame)

Here we are looking for the squares on the chessboard. It is done by using the findContours function from cv2 on the threshold image. The we then calculate the convex hull of the contour to get rid of any distortions. If the contour has 4 edges and the lengths of the opposite edges are similar, then the contour is considered a chessboard square. Information about the chessboard squares: their with, height and the slopes of the edges will be used in chessboard grid detection.

gap_lengths, new_lines = find_square_parameters(frame)
lines.extend(new_lines)

Based on the square parameters we try to find a subset of 18 lines representing a grid. Lines should be in the similar distances from each other as the length of the square edges and their slopes should be similar to square slopes.

chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.1)

Finally based on the chosen lines and their intersections we can define a chessboard grid with each cell being its own contour.

square_contours = find_square_contours(frame, chessboard_lines)

Chessboard tracking an figure detection

As on the hard videos the image is shaky the position of the chessboard must be tracked. To achieve this we find a bounding box by identifying bottom-left and top-right corners of the chessboard grid. The chessboard is then tracked by the MIL tracker. The grid is adjusted appropriately to the tracked bounding box.

The figures are found using a cv2 function goodFeaturesToTrack not to include chessboard corners we filer out those points that are close to grid lines.

Processing steps for each subset

Easy

Easy1

chess = video_from_path('recordings\Easy1.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.1, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)

Video loaded
912 720
30.00030000300003

write_and_detect_chessboard(chess, square_contours, 'Easy1')

Easy2

chess = video_from_path('recordings\Easy2.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.18, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)

Video loaded
920 720
30.00030000300003

write_and_detect_chessboard(chess, square_contours, 'Easy2')

Easy3

chess = video_from_path('recordings\Easy3.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.15, if_display=False)
chessboard_lines = [chessboard_lines[0], chessboard_lines[2]]
square_contours = find_square_contours(frame, chessboard_lines)

Video loaded
876 720
30.00030000300003

write_and_detect_chessboard(chess, square_contours, 'Easy3')

Medium

Medium 1

chess = video_from_path('recordings\Medium1.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.15, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)

Video loaded
866 720
30.00030000300003

write_and_detect_chessboard(chess, square_contours, 'Medium1')

Medium 2

chess = video_from_path('recordings\Medium2.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.15, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)

Video loaded
884 720
30.00030000300003

write_and_detect_chessboard(chess, square_contours, 'Medium2')

Medium 3

chess = video_from_path('recordings\Medium3.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.15, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)

Video loaded
876 720
30.00030000300003

write_and_detect_chessboard(chess, square_contours, 'Medium2')

Hard

Hard1

chess = video_from_path('recordings\Hard1.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.2, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)

Video loaded
816 720
30.00030000300003

write_and_detect_chessboard(chess, square_contours, 'Hard1')

Hard2

from utils import *

chess = video_from_path('recordings\Hard2.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.2, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)

Video loaded
798 720
30.00030000300003

write_and_detect_chessboard(chess, square_contours, 'Hard2')

Hard3

chess = video_from_path('recordings\Hard3.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.2, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)

Video loaded
720 752
30.00030000300003

write_and_detect_chessboard(chess, square_contours, 'Hard3')

Exemplary frames

Easy

Medium

Hard

Problems encountered so far.

For some of the videos the reflection of the light on the chessboard makes the point detection algorithm find many false positives in that region. We are trying to solve that problem with including more measures to the figure tracking process: histogram, brightness.

For some of the video frames. The tracking is not perfect thus resulting in slight problems in figure recognition.